Azure Storage

What is Azure Storage?

Azure Storage is Microsoft's cloud computing storage solution that provides highly scalable, durable, and available storage services for applications and data. It offers several types of storage services to cater to different needs.

Microsoft Azure has many data centers within different regions. Every region can have one or multiple Availability Zones and they are isolated from each other. Each zone can have one or more multiple data centers with independent power, cooling, and networking. Microsoft Azure Storage is a service capable of storing multiple copies of your data and provides scalable, secure, and highly available storage to meet diverse storage needs. Whether you require object storage, file storage, or structured data storage, Azure Storage supports them.

What is an Azure Storage Account?

Azure Storage is accessed through a storage account with a unique address. It serves as a unique namespace in Azure for your data objects, providing a way to access and manage different types of Azure Storage services within a single account. The storage account contains all your storage objects like blobs, tables, queues, and files, etc. These objects are secure, durable, highly available and can be accessed from anywhere over HTTP and HTTPS endpoints. When you create a storage account, you specify its settings such as replication options, access tiers, and networking configuration.

What types of Azure Storage Account are available? Which type of Azure Storage Account should I use?

Broadly, there are 2 types of Storage Accounts widely used:

General Purpose: Standard General-Purpose v2 (GPv2) is designed to operate with all types of Microsoft Azure Storage, except Disk Storage. To create disks inside your Azure Storage, you should first create the Microsoft Azure Virtual Machine. This type of account supersedes its predecessor – General-Purpose v1 (to be consistent with others).

Blobs: A Blob Storage account is designed to work with Block Blobs and Append Blobs. It’s best for high-transaction-rate scenarios and scenarios that use smaller objects or require consistent low storage latency. This type of storage account is optimized specifically for storing unstructured data such as blobs (binary large objects) - for example: images, videos, documents, logs, and backups. Blob Storage accounts offer lower pricing compared to GPv2 accounts for storing blob data.

A detailed overview of the different storage account types is given by Microsoft, here: Storage account overview - Azure Storage | Microsoft Learn.

Types of Azure Storage

Azure Storage offers several types of storage, each designed to handle different kinds of data, use cases and workloads. Let's take a closer look at the storage types supported by Azure Storage:

Blob Storage	Ideal for storing unstructured data such as documents, media files, Virtual Hard Disks (VHDs), database backups, application logs, and application installers. Think of it as your digital warehouse, scalable and cost-effective and allows direct access to individual objects. These are stored in containers which are like folders in Windows File System. It supports Cool, Cold, Hot and Archive access tiers, enabling you to optimize costs based on data access frequency.
File Storage	Provides fully managed file shares in the cloud that are accessible via the industry-standard Server Message Block (SMB) protocol allowing seamless access and integration with existing applications and services running on Windows, Linux and MacOS. It is an excellent choice for scenarios like file sharing, content management, and application migration.
Queue Storage	A service for storing large numbers of messages that can be accessed asynchronously from anywhere in the world.
Table Storage	A NoSQL datastore providing a key-attribute store with schema-less design making it suitable for scenarios like logging, telemetry, etc.
Disk Storage	Provides disks for virtual machines (managed and unmanaged disks), containers, applications, and other services to access and use as they would a physical disk in their servers. Imagine a super-fast hard drive in the cloud, delivering low latency and consistent performance.
Elastic SAN	A fully integrated solution that simplifies deploying, scaling, managing, and configuring a SAN in Azure.
Container Storage	A volume management, deployment, and orchestration service built natively for containers.
NetApp Files	Powered by NetApp, makes it easy for enterprise line-of-business (LOB) and storage professionals to migrate and run complex, file-based applications with no code change and managed via NetApp accounts accessed via NFS, SMB and dual-protocol volumes.

Where can I find details of Azure Storage pricing?

The Microsoft website carries up-to-date details of storage pricing, see: Pricing overview – How Azure pricing works | Microsoft Azure.

Can Azure Storage be integrated with other Azure services?

Yes, Azure Storage can be integrated with a wide range of other Azure services, enabling powerful solutions for various use cases. Here are some examples of how Azure Storage is frequently integrated with other Azure services:

Azure Virtual Machines: Azure Storage can be used to store the virtual hard disks (VHDs) of Azure Virtual Machines (VMs). Managed disks, which are backed by Azure Storage, provide durable and high-performance storage for VMs.
Azure App Service: Azure Storage can be used to store files, images, and other static content for web applications hosted on Azure App Service. Blob Storage is commonly used for this purpose.
Azure Functions: Azure Functions can trigger actions based on changes to data stored in Azure Storage. For example, you can create serverless functions that are triggered when new blobs are uploaded or when items are added to queues or tables.
Azure Data Factory: Azure Data Factory can be used to orchestrate and automate data movement and transformation workflows. Azure Storage serves as a data source and destination for Azure Data Factory pipelines.
Azure Cosmos DB: Azure Cosmos DB supports storing and querying JSON data in Azure Storage, allowing you to leverage Cosmos DB's globally distributed database service for applications that require low-latency access to large volumes of semi-structured data.
Azure Backup: Azure Backup can be used to back up data stored in Azure Storage, providing a reliable and scalable backup solution for virtual machines, files, and databases hosted in Azure.

An overview of some of Azure’s most popular services is available, here: Top Azure Cloud Services Explained | eG Innovations.

How can I leverage storage tiers to reduce the costs of using Azure Storage?

Most types of Azure Storage have several tiers available. To manage costs as your need for storage grows, it can be wise to organize your data based on how frequently it will be accessed and how long it needs to be retained. Azure storage offers different access tiers so that you can store your blob data in the most cost-effective manner based on how it's being used (see: Access tiers for blob data - Azure Storage | Microsoft Learn).

Consider a scenario where you have a large volume of data that you need to store in Azure Storage, but not all of that data needs to be accessed frequently. In this case, you can use different tiers of Azure Storage to reduce costs while still meeting your storage requirements.

Imagine you have a data analytics platform where you collect sensor data from IoT devices. This data is collected continuously, but only the most recent data (last 30 days) is actively queried and analyzed for real-time insights. However, you want to retain the historical data for compliance and future analysis purposes.

Here's how you could use different tiers of Azure Storage to optimize costs:

Hot Blob Storage: You can store the most recent data, which needs to be accessed frequently, in the Hot access tier of Azure Blob Storage. This data is readily available for fast access and is suitable for workloads that require frequent access.
Cool Blob Storage: Once the data becomes older than 30 days and is no longer actively queried, you can transition it to the Cool access tier of Azure Blob Storage. The Cool tier offers lower storage costs compared to the Hot tier but may have slightly higher access costs. Since this data is accessed infrequently, the Cool tier is a cost-effective option for storing it.
Archive Blob Storage: After a certain period (let's say after 1 year), the historical data becomes even less likely to be accessed but still needs to be retained for compliance reasons. You can then transition this data to the Archive access tier of Azure Blob Storage. The Archive tier offers the lowest storage costs but has higher access costs and longer retrieval times. This tier is ideal for long-term retention of data that is rarely accessed.

By leveraging the Hot, Cool, and Archive access tiers of Azure Blob Storage based on the access patterns and lifecycle of your data, you can significantly reduce storage costs while ensuring that your data remains accessible and compliant. Additionally, Azure Blob Storage offers lifecycle management policies that automate the transition of data between different access tiers based on user-defined rules, further streamlining cost optimization.

How should I monitor Azure Storage?

When using Azure Storage there are key questions related to the health, performance, and capacity of your Azure Storage that you will need to be able to answer and any monitoring solution will need to provide answers such as:

Are my Azure Storage services available?
Are the any Azure Storage accounts with no or low usage?
Are there any issues with the number of requests to the Azure Storage services?
Is my Azure Storage service experiencing latency?
Am I approaching the capacity limits of my Azure Storage account?
Are there any failures in the storage requests?
Are there any performance issues with the Azure Storage services?
How much storage capacity is currently being used?
Are there any network-related issues affecting data transfer to and from Azure Storage?
Are there any security breaches or unauthorized access attempts to my Azure Storage?
How is the data transfer rate between my applications and Azure Storage?
How much is my Azure Storage costing the organization? How will my storage needs grow and how much will that cost?

Key metrics for monitoring Azure Storage

Here are the details of some of the key health and performance metrics you will need to monitor when using Azure Storage.

Availability Metrics	Availability	Measures the percentage of time that Azure Storage is available for read and write operations.
Capacity Metrics	Used capacity, Allocated capacity	Used capacity tracks the amount of storage space currently utilized by your data. Allocated capacity measures the total amount of storage space allocated to your Azure Storage account.
Performance Metrics	Throughput, Transactions, Latency, Error rate	Throughput measures the amount of data transferred to and from Azure Storage within a specific time frame. Transactions tracks the number of operations performed on your Azure Storage account, including reads, writes, and deletes. Latency monitors the response time for read and write operations, helping identify potential performance bottlenecks.
Network Metrics	Ingress, Egress, Bandwidth	Ingress and Egress measures the amount of data transferred into and out of Azure Storage. Bandwidth tracks the rate at which data is transferred between your applications and Azure Storage.
Security Metrics	Authentication failures, Authorization errors	Authentication failures tracks the number of failed authentication attempts. Authorization errors monitors the number of access-related errors, such as unauthorized access attempts.

Can I Use Azure Monitor to monitor Azure Storage?

Azure Monitor is Microsoft’s native monitoring and provides a degree of monitoring for Azure. An overview of Azure Monitor is provided, here: What is Azure Monitor? - IT Glossary | eG Innovations. Many users find calculating the costs of using Azure Monitor somewhat complex, we’ve provided a guide and some links to help you get started with estimating your costs for using Azure Monitor, see: Estimating Azure Monitor Cost for AVD Monitoring (eginnovations.com).

Azure Monitor Insights helps to monitor all the key metrics mentioned above for Azure Storage accounts. But there are few challenges:

Azure Monitor Insights allows to monitor only 200 storage accounts, regardless of the number of subscriptions that are selected.
By default, there are no alerts configured for Azure Storage accounts. You need to create alerts for key metrics. Please look at this link for creating alerts for Azure File Storage service. Most admins would struggle to calculate a number to act as an alert threshold for a metric such as “Blob success end-to-end latency (Seconds)” on an Azure storage disk. Beyond the need for significant domain-knowledge to set up thresholds on most metrics, static metrics simply are not suitable for measures that vary with time or with server. Learn more about threshold calculation, here: Static vs Dynamic Alert Thresholds for Monitoring | eG Innovations.
Azure Monitor Insights show the error details [Show Logs feature] for the storage account alone. If you have many file shares and you want to look the error details for each file shares, it is not possible. This show log feature shows the first 250 rows alone and does not contain 1:1 mapping between errors in logs and transaction failures.
Azure Monitor does not allow to drill down and see the latest server side, client-side or authorization errors. It shows all errors and allow the user to find it on their own.
Azure Monitor does not show Storage account capacity utilization. You need to write a PowerShell script to do it.

With the help of eG Enterprise Azure Subscription monitoring, you can answer you Azure Storage questions easily. At the time of writing, the following metrics are reported for each Storage Account available in the Azure Subscription.

Overview Provisioning state Disk primary state Storage account service availability	Storage Capacity Total storage capacity Storage account used capacity Used file capacity Used blob capacity Used table capacity Used queue capacity Available storage capacity Storage capacity utilization	Overall Transaction Details Total requests in storage account Total ingress in storage account Total egress in storage account Success server total latency Success end-to-end total latency
File Share Details File shares count Files in the storage account Snapshots present in storage account File share capacity quota File share requests File share ingress File share egress File share success server latency File share success end-to-end latency File share availability	Table Details Table shares count Tables entity count Table requests Table ingress Table egress Table success server latency Table success end-to-end latency	Blob Container Details Blob containers count Blob objects stored Azure data lake capacity Blob requests Blob ingress Blob egress Blob success server latency Blob success end-to-end latency
Queue Details Queue shares count Queue messages count Queue requests Queue ingress Queue egress Queue success server latency Queue success end-to-end latency

Learn more about eG Enterprise monitoring for Azure Services including storage, here: Azure Cloud Monitoring Tools for IaaS, PaaS, SaaS (eginnovations.com).

Simple overviews of all key Azure Storage metrics are provided within the eG Enterprise console. AIOps driven alerting is configured out-of-the-box and configured to proactively alert the administrator of problems and/or anomalous usage.